Information processing apparatus, information processing method, and storage medium

ABSTRACT

A region of an object is separated from an image captured by an image capturing unit included in a plurality of image capturing units configured to capture an image of an object from a plurality of viewpoints, using a method corresponding to light applied to an image-capturing target region of the image capturing unit at a time of capturing the image. Then, the shape of the object is determined based on the region of the object separated by a separation unit from each of a plurality of images captured by the plurality of image capturing units.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an information processing apparatus, aninformation processing method, and a storage medium.

Description of the Related Art

There are conventionally-known methods for extracting an objectcontained in an image, such as a background difference method in whichan object is extracted by determining the difference between an imagecontaining the object and a background image of a background notcontaining the object. Such a method is used in, for example, acquiringthe shape of an object from a multi-viewpoint object region using avisual hull method. In object extraction using a method such as thebackground difference method, pixels that are different in color and/orluminance value from those in a background image are extracted as pixelscontaining an object. The object is extractable with high accuracy iflight applied to an entire scene including the object and background isuniform. However, in a case in which a change in an illuminationenvironment, etc. leads to a successive change in color of the object aswell as the background, the object extraction accuracy may decrease. Todeal with such an issue, Japanese Patent Application Laid-Open No.10-21408 discusses a technique that realizes object extraction that candeal with a case of a time-series change in pixel values by updating athreshold value in the object extraction based on the amount of changein an image.

An object region is separated from each of a plurality of imagescaptured respectively by a plurality of image capturing units configuredto image an object from different viewpoints, and the shape of theobject is determined using the object regions separated from theplurality of images. In an image capturing region of the image capturingunits, the lighting of a flash, a sudden change in illumination appliedby an illumination device, etc. can cause a sudden change in the lightapplied to the image capturing region of the image capturing units. Inthis case, it is difficult to separate the object from the imagescaptured by the image capturing units with high accuracy by thetechnique discussed in Japanese Patent Application Laid-Open No.10-21408. This makes it difficult to determine the object shape withhigh accuracy.

SUMMARY OF THE INVENTION

The present disclosure is directed to an information processingapparatus capable of determining the shape of an object with higheraccuracy even in a case in which light applied to the object changes.

According to an aspect of the present disclosure, an informationprocessing apparatus includes a separation unit configured to separate,from an image captured by an image capturing unit included in aplurality of image capturing units configured to capture an image of anobject from a plurality of viewpoints, a region of the object using amethod for light applied to an image-capturing target region of theimage capturing unit at a time of capturing the image, and adetermination unit configured to determine a shape of the object basedon the region of the object separated by the separation unit from eachof a plurality of images captured by the plurality of image capturingunits.

Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of a hardwareconfiguration of an information processing apparatus.

FIG. 2A is a diagram illustrating an example of a layout of imagecapturing apparatuses in a case of viewing an image-capturing targetobject region from an obliquely-upward direction, and FIG. 2B is adiagram illustrating an example of the layout of the image capturingapparatuses in a case of viewing the image-capturing target objectregion from directly above.

FIG. 3A is a diagram illustrating an example of a background image ofthe image-capturing target object region captured by an image capturingapparatus, FIG. 3B is a diagram illustrating an example of an image thatcontains target objects and is captured by an image capturing apparatuswithout involving flash by a camera, FIG. 3C is a diagram illustratingan example of an image that contains the objects and is captured by animage capturing apparatus involving flash by the camera, FIG. 3D is adiagram illustrating an example of an image indicating a result of abackground difference method performed on the image illustrated in FIG.3B using the background image illustrated in FIG. 3A, and FIG. 3E is adiagram illustrating an example of an image illustrating a result of thebackground difference method performed on the image illustrated in FIG.3C using the background image illustrated in FIG. 3A.

FIG. 4 is a block diagram illustrating an example of a functionalconfiguration of the information processing apparatus.

FIG. 5 is a flowchart illustrating an example of a process performed bythe information processing apparatus.

FIG. 6A is a diagram illustrating an example of an image of an objectshape determined based on the image illustrated in FIG. 3E, which isviewed from a virtual viewpoint, and FIG. 6B is a diagram illustratingan example of an image of an object shape determined based on the imageillustrated in FIG. 3D, which is viewed from a virtual viewpoint.

FIG. 7 is a flowchart illustrating an example of a process performed bythe information processing apparatus.

FIG. 8 is a flowchart illustrating an example of a process performed bythe information processing apparatus.

FIG. 9 is a diagram illustrating an example of a layout of the imagecapturing apparatuses.

FIG. 10A is a diagram illustrating an example of a background imageacquired by capturing an image of an image-capturing target region fromthe front side, FIG. 10B is a diagram illustrating an example of acaptured image of an object standing on a stage, FIG. 10C is a diagramillustrating an example of an image indicating a result of backgrounddifference processing on the image illustrated in FIG. 10B using thebackground image illustrated in FIG. 10A, FIG. 10D is a diagramillustrating an example of an image captured in a state in which theobject stands on the stage and light is applied to the object and thestage, FIG. 10E is a diagram illustrating an example of an imageindicating a result of background difference processing performed on theimage illustrated in FIG. 10D using the background image illustrated inFIG. 10A, FIG. 10F is a diagram illustrating another example of an imagecaptured in a state in which the object stands on the stage and light isapplied to the object and the stage, and FIG. 10G is a diagramillustrating another example of an image indicating a result of thebackground difference processing on the image illustrated in FIG. 10Fusing the background image illustrated in FIG. 10A.

FIG. 11 is a block diagram illustrating an example of a functionalconfiguration of the information processing apparatus.

FIG. 12 is a flowchart illustrating an example of a process performed bythe information processing apparatus.

FIG. 13 is an example of a table illustrating correspondence between theemission state of an illumination device and pixels of the imagecapturing apparatuses.

DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments of the present disclosure will bedescribed in detail below with reference to the drawings.

FIG. 1 is a block diagram illustrating an example of a hardwareconfiguration of an information processing apparatus 100 according to afirst exemplary embodiment. The information processing apparatus 100 isconnected to a plurality of image capturing apparatuses 108 having adifferent viewpoint from one another, separates an object region fromthe images acquired from the image capturing apparatuses 108, anddetermines an object shape based on the separated object region. Theinformation processing apparatus 100 is, for example, a personalcomputer, a server apparatus, a tablet apparatus, a smartphone, acomputer built in a device. Further, the information processingapparatus 100 can be an image capturing apparatus such as a networkcamera.

The information processing apparatus 100 includes a central processingunit (CPU) 101, a random-access memory (RAM) 102, a read-only memory(ROM) 103, a secondary storage apparatus 104, an input interface 105,and an output interface 106, all of which are communicably connectedwith one another via a system bus 107.

The CPU 101 is a central processing unit configured to control theinformation processing apparatus 100. The RAM 102 is a storage apparatusconfigured to function as a work area of the CPU 101 and a temporarydata storage location. The ROM 103 is a storage apparatus configured tostore programs and the like for controlling the respective components ofthe information processing apparatus 100. The CPU 101 executes a programstored in the ROM 103 using the RAM 102 as the work area andcomprehensively controls the respective components of the informationprocessing apparatus 100 via the system bus 107.

The secondary storage apparatus 104 is a storage apparatus, such as ahard disk drive (HDD), solid state drive (SSD), flash memory, or opticaldisk drive, configured to store various programs, various settinginformation, images acquired from the image capturing apparatuses 108.The CPU 101 is capable of writing data to the secondary storageapparatus 104 and reading data stored in the secondary storage apparatus104 via the system bus 107.

The input interface 105 is, for example, a universal serial bus (USB)interface or an Institute of Electrical and Electronics Engineers (IEEE)1394 serial bus interface and is used to input data, instructions, andthe like from an external apparatus to the information processingapparatus 100. The information processing apparatus 100 is connected toeach of the plurality of image capturing apparatuses 108, an externalstorage apparatus 109 via the input interface 105. The CPU 101 acquiresvarious data (e.g., images including still images and moving imagescaptured by the image capturing apparatuses 108, data such as parametersrelating to image capturing conditions of the image capturingapparatuses 108) from the image capturing apparatuses 108 via the inputinterface 105. Further, the CPU 101 acquires data from the externalstorage apparatus 109 (e.g., a storage medium such as a hard disk,memory card, compact flash (CF) card, secure digital (SD) card, or USBmemory) via the input interface 105.

The output interface 106 is a serial bus interface, such as a USBinterface or an IEEE 1394 serial bus interface, similar to the inputinterface 105 and is used to output information to an externalapparatus. Further, the output interface 106 can be, for example, avideo image output terminal such as a digital visual interface (DVI) anda high-definition multimedia interface (HDMI®). The informationprocessing apparatus 100 is connected to the external storage apparatus109 and a display apparatus 110 via the output interface 106. The CPU101 outputs a processed image to the display apparatus 110 (any type ofimage display device such as a liquid crystal display) via the outputinterface 106 to display the image.

The CPU 101 executes processing based on a program stored in the ROM 103or the secondary storage apparatus 104 to realize the functions of theinformation processing apparatus 100 described below with reference toFIGS. 4 and 11, the processes illustrated in the flowcharts in FIGS. 5,7, 8, and 12.

FIG. 2A illustrates an example of a layout of the plurality of imagecapturing apparatuses 108 when a region of image-capturing targetobjects is viewed from an obliquely-upward direction. Each of the imagecapturing apparatuses 108 is, for example, an image capturing apparatus(image capturing unit) such as a network camera or a monitoring camera.Each of the plurality of image capturing apparatuses 108 is arranged insuch a way as to capture an image of a predetermined region from adifferent viewpoint from the others. In the example illustrated in FIG.2A, this region is a stage where objects 201, which are two persons, arepresent.

FIG. 2B illustrates an example of a layout of the plurality of imagecapturing apparatuses 108 when the region of image-capturing targetobjects is viewed from directly above. In FIGS. 2A and 2B, the pluralityof image capturing apparatuses 108 is arranged in such a way as tosurround the stage and captures images of the objects 201. The objects201 are acting, singing, moving, or the like on the stage, and images ofthe objects 201 acting, singing, moving, or the like are captured by theimage capturing apparatuses 108.

A coordinate system 202 in FIGS. 2A and 2B is a coordinate system thatis determined in the region and used to specify the positions of theimage capturing apparatuses 108, and the like. For example, thepositions of the image capturing apparatuses 108 are specified ascoordinate values in the coordinate system 202. A camera 203 has a flashfunction and captures an image of the objects 201 moving in a systemindependent of the information processing apparatus 100, from a singleviewpoint.

Now, a possible problem, which can occur in a state illustrated in FIGS.2A and 2B, will be described. The camera 203 turns on flash at a time ofimage capturing in order to capture a clearer object image. While thescene in which persons as the objects are moving on the stage isdescribed as an example in FIGS. 2A and 2B, the processing according tothe present exemplary embodiment is also applicable to other scenes suchas a scene in which the image capturing apparatuses 108 are arranged insuch a way as to surround an object other than a person and reconstructsthe shape of the object.

The image in FIG. 3A is a background image of an image-capturing targetregion that is captured by one of the image capturing apparatuses 108.As used herein, the background image refers to an image of theimage-capturing target region of the image capturing apparatuses 108 inwhich there is no object.

The images in FIGS. 3B and 3C are images that are captured by the sameimage capturing apparatus 108 and contain the objects 201. At a time atwhich the image in FIG. 3B is captured, the camera 203 has not performedimage-capturing and the flash of the camera 203 has not yet turned on.At a time at which the image in FIG. 3C is captured, the camera 203performs flash image-capturing, and the objects 201 and part of thestage are illuminated by the flash of the camera 203 and thus appearbrighter.

The images in FIGS. 3D and 3E are respectively images indicating resultsof background difference processing on the images illustrated in FIGS.3B and 3C using the background image illustrated in FIG. 3A. From theimage in FIG. 3D, it is understood that only the objects 201 areseparated by the background difference method. However, from the imagein FIG. 3E, it is understood that the part of the stage is alsoextracted besides the objects 201 due to the effect of the flash lightof the camera 203.

In a case where a result of multi-viewpoint background differenceprocessing in which the objects 201 are separated without an effect ofthe flash as in the image in FIG. 3D is used as input data in thedetermination of the shapes of the objects 201 using a method such as avisual hull method, the shapes of the objects 201 are determined withhigh accuracy. However, in a case where a result of backgrounddifference processing in which a portion other than the objects 201 isalso separated due to an effect of the flash as in the image in FIG. 3Eis used as input data, it is difficult to determine the shapes of theobjects 201 with high accuracy. For example, the shape of the part ofthe object 201 below the knees is separated as an inaccurate shape(i.e., a single aggregate shape), so that it is difficult to acquire theaccurate shapes of the legs.

The processing of the information processing apparatus 100 according tothe present exemplary embodiment will be described with reference toFIGS. 4 and 5.

FIG. 4 is a block diagram illustrating an example of a functionalconfiguration of the information processing apparatus 100. Theinformation processing apparatus 100 includes a captured data groupacquisition unit 401, an illumination information acquisition unit 402,a separation method determination unit 403, a separation unit 404, aparameter acquisition unit 405, and a shape determination unit 406.

The captured data group acquisition unit 401 acquires, from therespective image capturing apparatuses 108, data on images capturedrespectively by the image capturing apparatuses 108.

The illumination information acquisition unit 402 acquires illuminationinformation based on the images acquired by the captured data groupacquisition unit 401. The illumination information indicates details ofhow light is applied to the image-capturing target region of the imagecapturing apparatuses 108.

The separation method determination unit 403 determines, based on theillumination information acquired by the illumination informationacquisition unit 402, a method for use in separating objects from theimage captured by each of the image capturing apparatuses 108.

The separation unit 404 separates an object region from the imagecaptured by each of the image capturing apparatuses 108 using the methoddetermined by the separation method determination unit 403.

The parameter acquisition unit 405 acquires, from each of the imagecapturing apparatuses 108 via the input interface 105, cameraparameters, such as an internal parameter, external parameter, anddistortion parameter, of each of the image capturing apparatuses 108.The internal parameter refers to a parameter such as a coordinate valueof an image center and a focal length of a camera lens. The externalparameter refers to a parameter that indicates the position andorientation of the camera. The distortion parameter refers to aparameter that indicates a lens distortion of the camera. The cameraparameters may be estimated by structure-from-motion based onmulti-viewpoint image data or may be calculated in advance bycalibration using a chart. Further, the parameter acquisition unit 405may acquire the camera parameters such as the internal parameter,external parameter, and distortion parameter of each of the imagecapturing apparatuses 108 from the secondary storage apparatus 104.

The shape determination unit 406 determines an object shape based on thecamera parameters of the respective image capturing apparatuses 108acquired by the parameter acquisition unit 405 and the object regionseparated by the separation unit 404.

A process performed by the functional units of the informationprocessing apparatus 100 will be described with reference to FIG. 5.FIG. 5 is a flowchart illustrating an example of processing performed bythe information processing apparatus 100.

In step S501, the captured data group acquisition unit 401 acquires, viathe input interface 105, temporally-consecutive multi-viewpoint imagedata groups captured by the plurality of image capturing apparatuses108. In the present exemplary embodiment, there are sixteen imagecapturing apparatuses 108. The image capturing apparatuses 108 eachcapture a 10-second moving image (600 frames (still images)) of 60 fpsat synchronized timings. Accordingly, the captured data groupacquisition unit 401 acquires image data on 600-by-16 images. Thecaptured data group acquisition unit 401 outputs the acquired image datato the illumination information acquisition unit 402 and the separationunit 404.

In step S502, the illumination information acquisition unit 402 acquiresillumination information, which is information about light applied tothe image-capturing target regions of the image capturing apparatuses108 at the time of capturing the frames, based on the image dataacquired from the captured data group acquisition unit 401. In thepresent exemplary embodiment, the illumination information is assumed tobe information indicating, for each of the plurality of image capturingapparatuses 108, whether the light applied to the image-capturing targetregions of the plurality of image capturing apparatuses 108 at the timeof capturing the frames contains momentarily-applied light such as acamera flash. A camera flash is an example of short-term light. Theshort-term light refers to light applied only for a period not longerthan a predetermined threshold time period, such as a camera flash,spark light, or thunder light. In the present exemplary embodiment, theimage-capturing target regions of the image capturing apparatuses 108may be illuminated by a camera flash.

The illumination information acquisition unit 402 identifies, for eachof the image capturing apparatuses 108, the frame that theimage-capturing target region is illuminated by the camera flash duringthe image capturing, among the frames captured by each of the imagecapturing apparatuses 108. Then, the illumination informationacquisition unit 402 acquires information about the identified frame(information that specifies the image capturing apparatus 108 thatcaptures the frame (e.g., identification information such as anidentifier of the image capturing apparatus 108) and frameidentification information (e.g., frame number)) as the illuminationinformation. In the present exemplary embodiment, the illuminationinformation acquisition unit 402 identifies a frame that satisfies thefollowing formulas (1) to (3) as a frame affected by the flash.

$\begin{matrix}{{{I\left( {f,u,v} \right)} - {I\left( {{f - 1},u,v} \right)}} \geq \alpha} & (1) \\{{{I\left( {{f^{\prime} - 1},u,v} \right)} - {I\left( {f^{\prime},u,v} \right)}} \geq \alpha} & (2) \\\left. \begin{matrix}{{f^{\prime} - f} \geq \beta} \\{{f^{\prime} - f} \leq \chi}\end{matrix} \right\} & (3)\end{matrix}$

In formulas (1) and (2), I(f, u, v) is the luminance value of the pixelat the coordinate (u, v) of the image of the frame f. In formulas (1),(2), and (3), f and f′ each indicate a frame number. The frame number isthe number of the frame in frames in sequential order. In formulas (1)and (2), u and v are coordinate values in the image. In formulas (1),(2), and (3), α, β, and χ are predetermined threshold values.

A frame to be an inspection target of whether the frame is affected bythe flash is specified as “frame f”. Formula (1) is a formula used todetect whether the difference in the luminance value at the coordinate(u, v) between a frame f−1 (previous frame captured immediately beforethe frame f) and the frame f captured by the same image capturingapparatus 108 is greater than or equal to a threshold value α. Theillumination information acquisition unit 402 compares the frames f andf−1 using formula (1) to thereby determine whether there is a steep risein luminance at the time point at which the frame f is captured, withrespect to the frame f−1.

A frame f′ is a frame that is captured by the same image capturingapparatus 108 after the frame f. Formula (2) is a formula used to detectwhether the difference in the luminance value at the coordinate (u, v)between a frame f′−1 (previous frame captured immediately before theframe f′) and the frame f′ captured by the same image capturingapparatus 108 is greater than or equal to a threshold value α. Theillumination information acquisition unit 402 compares the frames f′ andf′−1 using formula (2) to thereby determine whether there is a steeprise in luminance at the time point at which the frame f′ is captured,with respect to the frame f′−1.

If the illumination information acquisition unit 402 determines that,for example, the frame f satisfies formula (1), the illuminationinformation acquisition unit 402 initializes the frame f′ as a framef+1. Then, the illumination information acquisition unit 402 determineswhether the frame f′ satisfies formula (2), and if the illuminationinformation acquisition unit 402 determines that the frame f′ does notsatisfy formula (2), the illumination information acquisition unit 402updates the frame f′ by incrementing the frame number of the frame f′ byone. Specifically, in a case in which the frame f′ is the frame f+1, theframe f′ is updated to a frame f+2. Then, the illumination informationacquisition unit 402 re-determines whether the frame f′ satisfiesformula (2). The illumination information acquisition unit 402 repeatsthe foregoing processing until the frame f′ that satisfies formula (2)is found, thereby detecting the frame f′ that satisfies formula (2).

If the illumination information acquisition unit 402 detects the framef′ that satisfies formula (2), the illumination information acquisitionunit 402 determines whether the frames f and f′ satisfy formula (3)(i.e., whether the frame interval between the frames f and f′ is notless than β (e.g., 1, 2) and not more than χ (e.g., 5, 6)). Then, if theillumination information acquisition unit 402 determines that the framesf and f′ satisfy formula (3), the illumination information acquisitionunit 402 identifies the frames f, f+1, . . . , and f′−1 as the framesaffected by the flash.

The case in which formulas (1) to (3) are satisfied is a case in whichthe image-capturing target region is illuminated by light having anintensity not less than a predetermined intensity applied for a periodnot longer than the predetermined threshold period (e.g., period notlonger than the period corresponding to the χ-frame interval). Theillumination information acquisition unit 402 can identify such framesaffected by the light by detecting the frames f and f′ that satisfyformulas (1) to (3).

Then, the illumination information acquisition unit 402 associatesinformation indicating the image capturing apparatuses 108 correspondingto the frame identified as a frame affected by the flash and framenumber information about the frame, and outputs the illuminationinformation to the separation method determination unit 403 as theillumination information.

In the present exemplary embodiment, the illumination informationacquisition unit 402 performs the above-described processing to therebyidentify the frames affected by the flash, acquire the illuminationinformation, and output the illumination information. Alternatively, theillumination information acquisition unit 402 may, for example, performforeground background separation on all combinations of frames ofadjacent frame numbers using the background difference method andidentify a frame with a steep increase in foreground region size as aframe in which a change in luminance occurs due to the effect of theflash.

In step S503, the separation method determination unit 403 performscontrol so that all frames captured by the plurality of image capturingapparatuses 108 (all frames acquired in step S501) are subjected to theprocessing of steps S504 to S506. More specifically, the separationmethod determination unit 403 selects, from all the frames acquired instep S501, one frame to be a target of the processing of steps S504 toS506. Hereinafter, the frame that is selected will be referred to as“selected frame”.

Then, if the processing of steps S504 to S506 on the selected frame iscompleted, the separation method determination unit 403 selects a framethat has not been selected, as a new selected frame from all the framesacquired in step S501. The separation method determination unit 403repeats the above-described processing until the processing of stepsS504 to S506 on all the frames acquired in step S501 is completed.

In step S504, the separation method determination unit 403 determines amethod for separating an object region from each frame acquired by theimage capturing apparatuses 108, based on the illumination informationacquired in step S502. In the present exemplary embodiment, theillumination information includes the number of the image capturingapparatus that captures an image of the object illuminated by the flash,and the frame number of the captured frame. Thus, the separation methoddetermination unit 403 determines whether the illumination informationcontains information about the correspondence between the number of theimage capturing apparatus 108 corresponding to the selected frame andthe frame number corresponding to the selected frame.

If the separation method determination unit 403 determines that theillumination information contains information about the correspondencebetween the number of the image capturing apparatus 108 corresponding tothe selected frame and the frame number corresponding to the selectedframe (YES in step S504), the separation method determination unit 403determines the method of step S506 as the object separation method forthe selected frame, and the processing proceeds to step S506. On theother hand, if the separation method determination unit 403 determinesthat the illumination information does not contain information about thecorrespondence between the number of the image capturing apparatus 108corresponding to the selected frame and the frame number correspondingto the selected frame (NO in step S504), the separation methoddetermination unit 403 determines the method of step S505 as an objectseparation method for the selected frame, and the processing proceeds tostep S505.

In step S505, the separation unit 404 performs object region separationprocessing on the selected frame that is not affected by the flash. Inthe present exemplary embodiment, the separation unit 404 uses thebackground difference method to separate an object region from the framethat is not affected by the flash. The background difference method is amethod in which a background image not containing an object is comparedwith an image containing the object and a foreground object is extractedbased on a difference in color and/or luminance. With this processing,the background object and the object region can be separated. In thepresent exemplary embodiment, the separation unit 404 uses, as inputdata, the plurality of temporally-consecutive images captured by thesame image capturing apparatus 108 and acquired in step S501, andidentifies pixel values for each image of the plurality of images, andcalculates an intermediate value of the identified pixel values for eachpixel. Then, the separation unit 404 generates, as a background image,an image in which the pixel value of each pixel corresponds to thecalculated intermediate value.

Then, the separation unit 404 compares the pixel value of each pixel inthe generated background image with the pixel value of the correspondingpixel in the selected frame, and if the difference between the pixelvalues is not less than a threshold value, the separation unit 404separates the object in the pixel as an object region. In the presentexemplary embodiment, the separation unit 404 generates the backgroundimage to be used for the images captured by the image capturingapparatuses 108 based on the images captured by the image capturingapparatuses 108. Alternatively, the separation unit 404 may generate abackground image by capturing an image of the image-capturing targetregion without an object in advance using the image capturingapparatuses 108.

The separation unit 404 outputs information about the separated objectregion to the shape determination unit 406. The object regioninformation is, for example, a binary image (silhouette data) in which aregion with an object in the image is expressed white and a backgroundregion with no object is expressed black.

In step S506, the separation unit 404 performs object region separationprocessing on the selected frame affected by the flash. In the presentexemplary embodiment, as a result of the object region separationprocessing on the selected frame affected by the flash, the separationunit 404 directly uses the result of the object region separationprocessing performed on the frame of one frame before the selected framecaptured by the same image capturing apparatuses 108 as the selectedframe. There can be a case in which the frame of one frame before theselected frame is affected by the flash. However, the result of theobject region separation performed on the frame of one frame before theframe of one frame before the selected frame is applied to the result ofthe object region separation performed on the frame of one frame beforethe selected frame. As described above, the separation unit 404 can usethe result of the object separation processing of the object region inthe frame that is not affected by the flash, as result of the objectseparation processing of the object region in the frame that is affectedby the flash.

In a case in which an image is captured by a camera with a resolution of4096-by-2160 and a frame rate of 60 fps so that the vertical size of aperson as an object in the image is ⅛, since the moving speed of aperson is about 4 km/h, the object is moved by about 2.8 pixels betweenthe frames. The object is separated at the position shifted by 2.8pixels from the actual object position, but the shift of 2.8 pixels canbe considered as a small shift in the image of a resolution of4096-by-2160. Thus, the small shift of the object position between theadjacent frames can be ignored in the case in which the object isseparated from the temporally-consecutive frames (60 fps) and the shapeof the object is determined as in the present exemplary embodiment. Theseparation unit 404 outputs the result of the object region separationto the shape determination unit 406.

In the present exemplary embodiment, the separation unit 404 separatesthe object region from the frame affected by the flash in step S506using the above-described method. Alternatively, the separation unit 404may apply, as the result of the object region separation on the frameaffected by the flash, for example, a result of object region separationperformed on a frame that is not the frame of one frame before the frameaffected by the flash and is captured at a time point different from thetime point at which the frame affected by the flash is captured. Forexample, the separation unit 404 may apply, as the result of the objectregion separation performed on the frame affected by the flash, a resultof object region separation performed on a frame that is not affected bythe flash and is captured after the frame affected by the flash iscaptured. For example, in a case in which two consecutive frames areaffected by the flash, the separation unit 404 may perform operation asfollows. Specifically, the separation unit 404 may apply the result ofthe object region separation performed on the frame of one frame beforethe target frame for the first frame, and apply the result of the objectregion separation performed on the frame of one frame after the targetframe for the second frame.

Further, the separation unit 404 may generate a result of object regionseparation on the frame affected by the flash based on results of objectregion separation on a plurality of frames captured at a time pointdifferent from the time point at which the frame affected by the flashis captured. For example, in a case in which one frame is affected bythe flash, the separation unit 404 calculates an average of the resultof the object region separation on the frame of one frame before thetarget frame and the result of the object region separation on the frameof one frame after the target frame. Then, the separation unit 404 maydetermine the calculated average separation result as a result of objectregion separation on the target frame affected by the flash.

In step S507, the parameter acquisition unit 405 acquires the cameraparameters of the respective image capturing apparatuses 108 therefromvia the input interface 105 or from the secondary storage apparatus 104.The parameter acquisition unit 405 outputs the acquired cameraparameters to the shape determination unit 406.

In step S508, the shape determination unit 406 determines the shape ofthe object based on the object region information input from theseparation unit 404 in step S505 or S506 and the camera parameters inputfrom the parameter acquisition unit 405 in step S507. In the presentexemplary embodiment, the shape determination unit 406 determines theobject shape using the visual hull method. In the shape determinationusing the visual hull method, sampling points in a space where theobject may possibly exist are projected onto the object regionseparation result (silhouette data), and the object shape isreconstructed by inspecting whether the sampling points are commonlyincluded in the silhouette across a plurality of viewpoints. In thevisual hull method, highly-accurate shape reconstruction is possible byreducing the sampling width in the space. The shape determination unit406 outputs information about the determined object shape by, forexample, displaying the information on the display apparatus 110.Further, the shape determination unit 406 may output the informationabout the determined object shape by storing the information in thesecondary storage apparatus 104. Further, the shape determination unit406 may output the information about the determined object shape bytransmitting the information to a predetermined transmission destinationsuch as an external information processing apparatus.

In the present exemplary embodiment, the process is described in whichthe information processing apparatus 100 determines an object shape withhigher accuracy in the case in which the image-capturing target regionof the image capturing apparatuses 108 is illuminated by the cameraflash. The information processing apparatus 100 is also capable ofdetermining an object shape with high accuracy using a similar processeven in a case in which the image-capturing target region of the imagecapturing apparatuses 108 is illuminated by short-term light (e.g.,light of thunder, and spark) different from the camera flash.

As described above, in the present exemplary embodiment, the informationprocessing apparatus 100 separates an object region from the imagescaptured by the plurality of image capturing apparatuses 108 using amethod suitable for the light applied to the image-capturing targetregion when the images are captured. Then, the information processingapparatus 100 determines the object shape based on the separated objectregion. In this way, the information processing apparatus 100 candetermine the object shape with high accuracy even in the case in whichthe light applied to the object imaged by the image capturingapparatuses 108 is changed suddenly.

As described above with reference to FIGS. 3A to 3E, if the backgrounddifference method is used to separate an object from a frame affected bythe flash, a region other than the object is also separated erroneouslyas illustrated in FIG. 3E. FIG. 6A illustrates an example of a state inwhich the object shape determined based on the separation result isviewed from a virtual viewpoint. It is determined that a portion of theobject is in an area although there is nothing in the area, as an object601 in FIG. 6A, and the shape of the stage on which the object isstanding appears to be distorted. This occurs because erroneous resultsof object separation are acquired by a plurality of cameras affected bythe flash and the shape is determined using the erroneous results. Theshape and size of the object 601 vary depending on the performance,position, and other conditions of the camera that captures images withthe flash being lit. For example, even if the shape acquisition isperformed on all the frames captured temporally consecutively and theshape is viewed as a temporally-consecutive shape, since the shapeacquired while the flash is lit is distorted, the viewer is likely tofeel very strange.

In the present exemplary embodiment, the information processingapparatus 100 determines the object shape using the object regionseparated from the image that is not affected by the flash as in FIG. 3Dinstead of the image illustrated in FIG. 3E. FIG. 6B illustrates anexample of a state in which the determined object shape is viewed fromthe same viewpoint as in FIG. 6A. It is understood that the object 601,which appears in FIG. 6A but does not exist, disappears.

In the first exemplary embodiment, the process is described in which theobject shape is determined with high accuracy even at the timing atwhich the flash is lit, by acquiring the result of the object separationon the frame affected by the flash based on a result of objectseparation on another frame of the same camera.

However, as described above in step S506 in FIG. 5, in the case of amoving object, a small error occurs in a result of object separation. Ina second exemplary embodiment, a process for reducing an error that isgenerated due to the movement of an object will be described below.

The hardware configuration of the information processing apparatus 100according to the present exemplary embodiment is similar to that in thefirst exemplary embodiment.

The functional configuration of the information processing apparatus 100according to the present exemplary embodiment includes the captured datagroup acquisition unit 401, the illumination information acquisitionunit 402, the separation method determination unit 403, the separationunit 404, the parameter acquisition unit 405, and the shapedetermination unit 406, as in the first exemplary embodiment. However,the illumination information acquisition unit 402 is configured toacquire different illumination information from that in the firstexemplary embodiment. Further, details of the processing performed bythe separation unit 404 are different from those in the first exemplaryembodiment.

A process in the present exemplary embodiment will be described focusingon the difference between the present exemplary embodiment and the firstexemplary embodiment.

FIG. 7 is a flowchart illustrating an example of a process performed bythe information processing apparatus 100 according to the presentexemplary embodiment. In step S801, the captured data group acquisitionunit 401 acquires, via the input interface 105, a temporally-consecutivemulti-viewpoint image data group captured by the plurality of imagecapturing apparatuses 108. The captured data group acquisition unit 401outputs the acquired image data to the illumination informationacquisition unit 402 and the separation unit 404.

In step S802, the illumination information acquisition unit 402 acquiresillumination information. In the present exemplary embodiment, theillumination information acquisition unit 402 acquires illuminationinformation containing information specifying the image capturingapparatuses 108, information about the frame affected by the flash(e.g., frame number), and information about the coordinate value of thepixel affected by the flash, which are associated with one another. Thecoordinate value of the pixel affected by the flash is, for example,information about u and v that satisfy formulas (1) to (3).

In step S803, the parameter acquisition unit 405 acquires the cameraparameters of the respective image capturing apparatuses 108 therefromvia the input interface 105 or from the secondary storage apparatus 104.The parameter acquisition unit 405 outputs the acquired cameraparameters to the shape determination unit 406.

In step S804, the separation method determination unit 403 performscontrol so that all frames captured by the plurality of image capturingapparatuses 108 (all frames acquired in step S801) undergoes theprocessing of steps S805 to S807. More specifically, the separationmethod determination unit 403 selects, from all the frames acquired instep S801, one frame to undergo the processing of steps S805 to S807.Hereinafter, the frame that is selected will be referred to as “selectedframe”. Then, if the processing of steps S805 to S807 on the selectedframe is completed, the separation method determination unit 403 selectsa frame that has not been selected, as a new selected frame from all theframes acquired in step S801. The separation method determination unit403 repeats the foregoing processing until the processing of steps S805to S807 is completed on all the frames acquired in step S801.

In step S805, the separation method determination unit 403 determines amethod for separating an object region from the selected frame based onthe illumination information acquired in step S802. The separationmethod determination unit 403 determines whether the illuminationinformation contains information about the correspondence between thenumber of the image capturing apparatus 108 corresponding to theselected frame and the frame number corresponding to the selected frame.

If the separation method determination unit 403 determines that theillumination information contains information about the correspondencebetween the number of the image capturing apparatus 108 corresponding tothe selected frame and the frame number corresponding to the selectedframe (YES in step S805), the separation method determination unit 403determines the method of step S807 as the object separation method forthe selected frame, and the processing proceeds to step S807. On theother hand, if the separation method determination unit 403 determinesthat the illumination information does not contain information about thecorrespondence between the number of the image capturing apparatus 108corresponding to the selected frame and the frame number correspondingto the selected frame (NO in step S805), the separation methoddetermination unit 403 determines the method of step S806 as an objectseparation method for the selected frame, and the processing proceeds tostep S806.

In step S806, the separation unit 404 performs object region separationprocessing on the selected frame that is not affected by the flash usingthe background difference method, as in step S505.

In step S807, the separation unit 404 performs object region separationprocessing on the selected frame that is affected by the flash. FIG. 8illustrates details of the processing performed in step S807. FIG. 8 isa flowchart illustrating an example of processing performed in stepS807.

In step S901, the separation unit 404 acquires the camera parametersthat has acquired in step S803.

In step S902, the separation unit 404 calculates the depth of each pixelaffected by the flash in the selected frame (pixel at the coordinatevalue (u, v) in the selected frame that is determined to satisfyformulas (1) to (3)) using the camera parameters acquired in step S901.As used herein, the depth of a pixel refers to the distance between theimage capturing apparatus 108 that captures the selected frame and theobject in the pixel. The distance between the image capturing apparatus108 that captures the selected frame and the object in the pixel is, forexample, the distance between the position in an image sensor of theimage capturing apparatus 108 that corresponds to the pixel and theobject.

The separation unit 404 calculates the depth of each pixel affected bythe flash in the selected frame based on the camera parameters acquiredin step S901 using a method such as a plane sweep method in which avirtual layer is set and the depth of the object is estimated as instereo matching.

In step S903, the separation unit 404 calculates, using the plane sweepmethod, or other methods, the depth of each pixel in the frame of oneframe before the selected frame that has the same coordinate as thecoordinate of a pixel affected by the flash in the selected frame, andthe depth of neighborhood pixels neighboring the pixel. The neighborhoodpixels neighboring the pixel are pixels located around the pixel andare, for example, pixels contained in a region (e.g., rectangular regionof 11-by-11 pixels, rectangular region of 9-by-9 pixels) of apredetermined size with the pixel being the center.

In a case in which the image capturing apparatuses 108 each acquire thedistance between the object and the image capturing apparatus 108 thatcorresponds to a pixel using a range sensor at a time of imagecapturing, the separation unit 404 may acquire the distances that havebeen acquired by the image capturing apparatuses 108 as the depth of apixel in steps S902 and S903.

In step S904, the separation unit 404 performs object separation on thepixels affected by the flash in the selected frame, based on thedistances acquired in steps S902 and S903. The separation unit 404performs the following processing on each pixel affected by the flash.

Specifically, with respect to a pixel having the same coordinate as thecoordinate of a pixel affected by the flash and the neighborhood pixels,the separation unit 404 compares the distances each acquired from theframe of one frame before the selected frame in step S903 with thedistances acquired for the pixels in the selected frame in step S902.Then, the separation unit 404 identifies the closest distance to thedistance acquired for the pixel in the selected frame in step S902,among the distances acquired for the pixel and the neighborhood pixelsfrom the frame one frame before the selected frame in step S903. Theseparation unit 404 determines the pixel in the frame of one framebefore the selected frame that corresponds to the identified distance,as the pixel of the same captured object as that of the pixel affectedby the flash in the selected frame.

Then, if the pixel in the frame of one frame before the selected framethat corresponds to the identified distance is an object region, theseparation unit 404 determines the pixel affected by the flash in theselected frame as the object region. Further, if the pixel in the frameof one frame before the selected frame that corresponds to theidentified distance is not an object but a background region, theseparation unit 404 determines the pixel affected by the flash in theselected frame as the background region.

By the foregoing processing, the separation unit 404 can more accuratelydetermine whether an object corresponding to a pixel affected by theflash in the selected frame is separated as an object in the frame oneframe before. In this way, the separation unit 404 can accuratelydetermine whether each pixel affected by the flash in the selected frameis an object region.

Further, the separation unit 404 uses the background difference methodto separate an object region regarding each pixel in the selected framethat is not affected by the flash, as in step S806.

In step S808, the shape determination unit 406 determines the objectshape based on the information about the object regions separated by theseparation unit 404 in step S806 or S807 and the camera parametersacquired by the parameter acquisition unit 405 in step S803. In thepresent exemplary embodiment, the shape determination unit 406determines the object shape using the visual hull method.

The shape determination unit 406 outputs information about thedetermined object shape by, for example, displaying the information onthe display apparatus 110. Further, the shape determination unit 406 mayoutput the information about the determined object shape by storing theinformation in the secondary storage apparatus 104. Further, the shapedetermination unit 406 may output the information about the determinedobject shape by transmitting the information to a predeterminedtransmission destination such as an external information processingapparatus

As described above, in the present exemplary embodiment, the informationprocessing apparatus 100 performs object separation on the pixelsaffected by the flash using the depths of the pixels affected by theflash. In this way, the information processing apparatus 100 canseparate the object regions more accurately than in the first exemplaryembodiment and, thus, can determine the object shape more accurately.

In a third exemplary embodiment, a process will be described below thatis suitable for a situation of illumination that changes intricately,such as an illumination in concert or live performance.

The hardware configuration of the information processing apparatus 100according to the present exemplary embodiment is similar to that in thefirst exemplary embodiment. Details of the functional configuration ofthe information processing apparatus 100 according to the presentexemplary embodiment will be described below with reference to FIG. 11.

A process in the present exemplary embodiment will be described,focusing on the difference between the present exemplary embodiment andthe first exemplary embodiment.

FIG. 9 illustrates an example of a layout of the plurality of imagecapturing apparatuses 108 according to the present exemplary embodiment.A coordinate system 1001 is a coordinate system that is determined inadvance for the image-capturing target region. The shape of the stage,the positions and orientations of the image capturing apparatuses 108,the positions and orientations of illumination devices 1002 and 1003 inan image-capturing scene are each specified as a coordinate or vector inthe coordinate system 1001. The illumination devices 1002 and 1003 applylight to the stage and have a time code synchronized with the imagecapturing apparatuses 108. Hereinafter, the light applied by theillumination devices 1002 and 1003 will be referred to as illuminationlight. Further, the irradiation angle of the illumination light,orientation, color of the illumination light of the illumination devices1002 and 1003 are controlled based on an instruction from an externalapparatus such as the information processing apparatus 100.

An image in FIG. 10A is a background image acquired by capturing animage of the image-capturing target region of the plurality of imagecapturing apparatuses 108 according to the present exemplary embodimentfrom the front. The background image may be an image captured in a stateof no object or no illumination, or may be an image generated byacquiring a mean pixel value or intermediate value of pixels of aplurality of frames captured temporally consecutively.

An image in FIG. 10B is an image acquired by capturing an image of astate in which an object stands on the stage, and an image in FIG. 10Cis an image showing a result of object region separation obtained byapplying the background difference method to the image in FIG. 10B usingthe background image in FIG. 10A. From the image in FIG. 10C, it isunderstood that only the object is separated.

FIG. 10D is a captured image of a state in which the object and thestage are irradiated with the illumination light from the illuminationdevice 1002 or 1003, and FIG. 10E is an image illustrating a result ofobject region separation processing on the image in FIG. 10D. FIG. 10Fis a captured image of a state in which the object and the stage areilluminated by the illumination light from the illumination device 1002or 1003, and FIG. 10G is an image illustrating a result of object regionseparation processing on the image in FIG. 10F.

From the images in FIGS. 10E and 10G, it is understood that not only theobject but also the region (illuminated region) on the stage irradiatedwith the illumination light by the illumination device 1002 or 1003 areseparated as an object. Especially in the example in FIG. 10G, theobject overlaps the illuminated region on the stage illuminated with theillumination light by the illumination device 1002 or 1003 and, thus,the boundary between the object and the illuminated region on the stageis unclear. If the object shape is determined using such an objectseparation result, an object shape that does not exist appears besidesthe object, as in FIG. 6A. The process in the present exemplaryembodiment is a process suitable for such a situation.

The process according to the present exemplary embodiment will bedescribed with reference to FIGS. 11 and 12.

FIG. 11 is a block diagram illustrating an example of a functionalconfiguration of the information processing apparatus 100 according tothe present exemplary embodiment. The information processing apparatus100 according to the present exemplary embodiment includes the captureddata group acquisition unit 401, the illumination informationacquisition unit 402, the separation method determination unit 403, theseparation unit 404, the parameter acquisition unit 405, and the shapedetermination unit 406, as in FIG. 4. Further, the informationprocessing apparatus 100 includes a scene information acquisition unit1201, which is different from FIG. 4.

The scene information acquisition unit 1201 acquires stage shapeinformation indicating the shape of the stage from the image capturingapparatuses 108 via the input interface 105 or from the secondarystorage apparatus 104. In the present exemplary embodiment, the imagecapturing apparatuses 108 can identify the stage shape by measuring thestage using the range sensor.

Further, the illumination information acquisition unit 402 in thepresent exemplary embodiment acquires the illumination information notbased on the image data input from the captured data group acquisitionunit 401 but from the illumination devices 1002 and 1003 via the inputinterface 105 or from the secondary storage apparatus 104.

Further, the separation method determination unit 403 according to thepresent exemplary embodiment receives the camera parameters input fromthe parameter acquisition unit 405.

In step S1301, the captured data group acquisition unit 401 acquiresimages captured by the image capturing apparatuses 108 and outputs theacquired images to the separation unit 404.

In step S1302, the illumination information acquisition unit 402acquires illumination information from the illumination devices 1002 and1003 via the input interface 105 or from the secondary storage apparatus104. In the present exemplary embodiment, the illumination informationcontains information about the positions and orientations of theillumination devices 1002 and 1003 at the time code synchronized withthe image capturing apparatuses 108, the irradiation angle of theillumination light, and the color of the illumination light, which areassociated with one another. The illumination information acquisitionunit 402 outputs the acquired illumination information to the separationmethod determination unit 403.

In step S1303, the parameter acquisition unit 405 acquires cameraparameters of the image capturing apparatuses 108 and outputs theacquired camera parameters to the separation method determination unit403 and the shape determination unit 406.

In step S1304, the scene information acquisition unit 1201 acquiresshape information indicating the shape of the stage in theimage-capturing target region from the image capturing apparatuses 108via the input interface 105 or from the secondary storage apparatus 104.The scene information acquisition unit 1201 outputs the acquired shapeinformation to the separation method determination unit 403.

In step S1305, the separation method determination unit 403 generates atable described below based on the illumination information acquired instep S1302, the camera parameters acquired in step S1303, and the shapeinformation acquired in step S1304. Specifically, the separation methoddetermination unit 403 generates a table that specifies how pixels ineach frame (time code) captured by the image capturing apparatuses 108is affected by the illumination light. An example of the table will bedescribed below with reference to FIG. 13.

The separation method determination unit 403 can find out the states(position and orientation, irradiation angle, color information) of theillumination devices 1002 and 1003 at each predetermined time pointbased on the illumination information. The separation methoddetermination unit 403 can determine a region on the stage that isilluminated by the illumination light from the illumination devices 1002and 1003 using, for example, formulas (4) to (6) below, based on thestates of the illumination devices 1002 and 1003 and the shapeinformation.

Formula (4) is a formula that expresses a vector of the illuminationlight applied by the illumination devices 1002 and 1003, in the lightray direction. Formula (5) is a formula that expresses the stage shape.In the present exemplary embodiment, the stage has a flat surface.Formula (6) is a formula for obtaining an intersection point x_(s)(three-dimensional coordinate) of the vector of the light ray directionand the flat surface that are obtained from formulas (4) and (5). Informulas (4) to (6), x is a vector indicating the direction of theillumination light, x₀ is a position coordinate of the illumination, sis a predetermined coefficient indicating the size of the vector x, p isa vector indicating the orientation of the illumination device(direction of the illumination light applied by the illuminationdevice), n is a normal vector of the flat surface of the stage, and q isa position coordinate of a single point on the flat surface of thestage.

The separation method determination unit 403 can calculate, usingformulas (4) to (6), the position on the stage at which the center ofthe illumination light from the illumination device 1002, 1003intersects. Further, the separation method determination unit 403 canidentify the position at which an end portion of the illumination lightfrom the illumination devices 1002 and 1003 intersects with the stage,by substituting not the vector p but a vector p′, which is obtained byadding a rotation corresponding to the irradiation angle of theillumination devices 1002 and 1003 to the vector p, into formula (6).Further, the separation method determination unit 403 can identify theposition at which light contained in the illumination light from theillumination devices 1002 and 1003 intersects with the stage, bysubstituting a vector obtained by adding a rotation corresponding to anangle smaller than the irradiation angle of the illumination light ofthe illumination devices 1002 and 1003 for the vector p into formula(6).

$\begin{matrix}{x = {x_{0} + {sp}}} & (4) \\{{n \cdot \left( {x - q} \right)} = 0} & (5) \\{x_{s} = {x_{0} - {\frac{n \cdot \left( {x - q} \right)}{n \cdot p}p}}} & (6)\end{matrix}$

Further, the separation method determination unit 403 can calculate theilluminated region of each camera of the image capturing apparatuses 108on the image using, for example, formula (7) based on the illuminatedregion on the stage and the camera parameters. In formula (7), “A” is aninternal matrix obtained from the internal parameter of the camera, u isa position corresponding to the point x_(s) in the image captured by theimage capturing apparatuses 108, R is an external matrix obtained fromthe external parameter, and t is a camera position.

u=A _(c) ⁻¹ R _(c) ⁻¹(x _(s) −t)  (7)

The separation method determination unit 403 can determine the regionilluminated by the illumination light on the image captured by the imagecapturing apparatuses 108 at a predetermined time point using formulas(4) to (7). The separation method determination unit 403 generates atable as illustrated in FIG. 13 based on the determined region. Thetable specifies which illumination is applied to each pixel of an imagecaptured by the image capturing apparatuses 108 at each predeterminedtime point based on the illumination information. The separation methoddetermination unit 403 stores the generated table in, for example, thesecondary storage apparatus 104.

How each pixel of an image captured by the image capturing apparatuses108 is affected by the illumination light in a state in which there isno object on the stage at a predetermined time point can be determinedby referring to the table stored in the secondary storage apparatus 104.In the example in FIG. 13, it is understood that, for example, the pixelof the coordinate value (1, 1) of the image captured by the imagecapturing apparatus 108 of the camera number 1 at a time pointcorresponding to the frame number 1 is affected by the illuminationdevice 1003 (blue illumination light). Further, it is understood thatthe pixel of the coordinate value (1, 1) of an image captured by theimage capturing apparatus 108 of the camera number 2 at a time pointcorresponding to the frame number 1 is affected by both the illuminationdevice 1002 (yellow illumination light) and the illumination device 1003(red illumination light). The separation method determination unit 403outputs the generated table as illustrated in FIG. 13 to the separationunit 404.

In step S1306, the separation method determination unit 403 performscontrol so that all frames captured by the plurality of image capturingapparatuses 108 (all frames acquired in step S1301) undergoes theprocessing of steps S1307 and S1308. More specifically, the separationmethod determination unit 403 selects, from all the frames acquired instep S1301, one frame to undergo the processing of steps S1307 andS1308. Hereinafter, the frame that is selected will be referred to as“selected frame”.

Then, if the processing of steps S1307 and S1308 on the selected frameis completed, the separation method determination unit 403 selects aframe that has not been selected, as a new selected frame from all theframes acquired in step S1301. The separation method determination unit403 repeats the foregoing processing until the processing of steps S1307and S1308 on all the frames acquired in step S1301 is completed.

In step S1307, the separation unit 404 generates, based on the tablegenerated in step S1305, a background image for use in the backgrounddifference method with respect to an image captured by the imagecapturing apparatuses 108 at a predetermined time point. The separationunit 404 refers to the table generated in step S1305 and acquires thecamera number of the camera that generates the background image, theframe number f, and the illumination state (applied illumination andapplied color) of the coordinate value (u, v). Then, the separation unit404 searches the table for a frame number corresponding to the sameillumination state as the acquired illumination state with the samecamera number c and the same coordinate value (u, v).

The separation unit 404 searches for all frames of the same illuminationstate, calculates, for example, an average pixel value or intermediatevalue of the coordinates (u, v) of the plurality of searched frames,determines the pixel value at the coordinate (u, v) of the backgroundimage. The separation unit 404 performs the foregoing processing on allpixels of the image to thereby generate the background image.

In step S1308, the separation unit 404 performs background differenceprocessing on the image captured by the image capturing apparatus 108corresponding to the background image at the time point corresponding tothe background image, using the background image generated in stepS1307, and separates the object. The separation unit 404 outputs theobject separation result to the shape determination unit 406.

In step S1309, the shape determination unit 406 determines the objectshape based on the information about the object region separated by theseparation unit 404 in step S1308 and the camera parameters acquired instep S1303 by the parameter acquisition unit 405. In the presentexemplary embodiment, the shape determination unit 406 uses the visualhull method to determine the object shape.

As described above, in the present exemplary embodiment, the informationprocessing apparatus 100 identifies a position illuminated by theillumination light based on the illumination information, the shapeinformation, and the camera parameters, and generates the table thatspecifies how each pixel of an image captured at each time point by theimage capturing apparatuses 108 is affected by the illumination light.Then, the information processing apparatus 100 generates, based on thegenerated table, a background image corresponding to the illuminationstate in each image capturing apparatus 108 and each frame, and uses thebackground image in the background difference method in the objectseparation. In this way, the information processing apparatus 100 canaccurately determine the object shape even in a case in which light thatchanges intricately, such as an illumination in concert or liveperformance, is applied to the image-capturing target region.

Other Exemplary Embodiment

An exemplary embodiment of the present invention is also realizable by aprocess in which a program for implementing one or more functions of theabove-described exemplary embodiments is supplied to a system orapparatus via a network or storage medium and one or more processors ofa computer of the system or apparatus read and execute the program.Further, an exemplary embodiment of the present invention is alsoimplemented by a circuit (e.g., application-specific integrated circuit(ASIC)) for implementing one or more functions.

For example, part of afunctional configuration of the informationprocessing apparatus 100 or an entire functional configuration may beimplemented as hardware in the information processing apparatus 100.

While the exemplary embodiments of the present invention have beendescribed in detail above, it should be apparent that the presentinvention is not to be limited by any specific exemplary embodiment. Theabove-described exemplary embodiments can be combined in any manner.

According to an exemplary embodiment of the present invention, an objectshape can be determined with high accuracy even in a case in which lightapplied to an image-capturing target region is changed.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2018-071723, filed Apr. 3, 2018, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An information processing apparatus, comprising: a separation unit configured to separate, from an image captured by an image capturing unit included in a plurality of image capturing units configured to capture an image of an object from a plurality of viewpoints, a region of the object using a method for light applied to an image-capturing target region of the image capturing unit at a time of capturing the image; and a determination unit configured to determine a shape of the object based on the region of the object separated by the separation unit from each of a plurality of images captured by the plurality of image capturing units.
 2. The information processing apparatus according to claim 1, wherein the separation unit separates the region of the object from the image captured by the image capturing unit included in the plurality of image capturing units using a background difference method corresponding to the light applied to the image-capturing target region of the image capturing unit at the time of capturing the image.
 3. The information processing apparatus according to claim 1, wherein, the separation unit separates, in a case where the light applied to the image-capturing target region of the image capturing unit included in the plurality of image capturing units at the time of capturing the image by the image capturing unit contains short-term light, which is light applied only for a period equal to or less than a predetermined threshold value, the region of the object from the image based on a region of the object separated from an image captured by the image capturing unit at a second time point, which is different from a first time point at which the image is captured, at which the short-term light is not applied to the image-capturing target region of the image capturing unit.
 4. The information processing apparatus according to claim 3, wherein the separation unit separates, in the case where the light applied to the image-capturing target region of the image capturing unit included in the plurality of image capturing units at the time of capturing the image by the image capturing unit contains the short-term light, from the image as the region of the object in the image, the region of the object in the image that is separated from the image captured at the second time point by the image capturing unit.
 5. The information processing apparatus according to claim 3, wherein the separation unit separates, in the case where the light applied to the image-capturing target region of the image capturing unit included in the plurality of image capturing units at the time of capturing the image by the image capturing unit contains the short-term light, from the image, the region of the object based on a plurality of the regions of the object in the image that is separated from each of a plurality of the images captured by the image capturing unit at a plurality of the second time points.
 6. The information processing apparatus according to claim 3, wherein the separation unit separates, in the case where the light applied to the image-capturing target region of the image capturing unit included in the plurality of image capturing units at the time of capturing the image by the image capturing unit contains the short-term light, from the image, the region of the object based on the region of the object separated from the image captured by the image capturing unit at the second time point and a distance between the region illuminated by the short-term light at the first time point and the image capturing unit.
 7. The information processing apparatus according to claim 1, wherein the separation unit separates the region of the object from the image captured by the image capturing unit included in the plurality of image capturing units using a background difference method that uses a background image corresponding to the light applied to the image-capturing target region of the image capturing unit at the time of capturing the image.
 8. The information processing apparatus according to claim 1, wherein the separation unit separates, from the image captured by the image capturing unit included in the plurality of image capturing units, the region of the object using a method corresponding to the light applied to the image-capturing target region of the image capturing unit at the time of capturing the image, based on a table specifying an effect of the light applied to the image-capturing target region of the image capturing unit at the time of capturing the image, on each pixel.
 9. The information processing apparatus according to claim 1, wherein the separation unit separates, from the image captured by the image capturing unit included in the plurality of image capturing units, the region of the object using a method determined based on illumination information, which is information about the light applied to the image-capturing target region of the image capturing unit.
 10. The information processing apparatus according to claim 9, wherein the illumination information contains information about a time point at which the light is applied and information indicating an image capturing unit that captures an image of a region illuminated by the light.
 11. The information processing apparatus according to claim 9, wherein the illumination information contains information indicating a region affected by the light applied to the image-capturing target region of the image capturing unit in the image captured by the image capturing unit included in the plurality of image capturing units.
 12. The information processing apparatus according to claim 9, wherein the illumination information contains information about a timing at which an illumination device configured to emit light emits the light, information indicating a position of the illumination device, information indicating a direction in which the illumination device emits the light, and information indicating a color of the light to be emitted by the illumination device.
 13. An information processing method to be executed by an information processing apparatus, the method comprising: separating, from a plurality of images captured by a plurality of image capturing units each configured to capture an image of an object from a plurality of viewpoints, a region of the object using a method corresponding to light applied to an image-capturing target region of each of the plurality of image capturing units at a time of capturing the image; and determining a shape of the object based on the region of the object separated from each of the plurality of images captured by the plurality of image capturing units.
 14. The information processing method according to claim 13, wherein the region of the object is separated from the image captured by the image capturing unit included in the plurality of image capturing units using a background difference method corresponding to the light applied to the image-capturing target region of the image capturing unit at the time of capturing the image.
 15. The information processing method according to claim 13, wherein the region of the object from the image, in a case where the light applied to the image-capturing target region of the image capturing unit included in the plurality of image capturing units at the time of capturing the image by the image capturing unit contains short-term light, which is light applied only for a period equal to or less than a predetermined threshold value, based on a region of the object separated from an image captured by the image capturing unit at a second time point, which is different from a first time point at which the image is captured, at which the short-term light is not applied to the image-capturing target region of the image capturing unit.
 16. A non-transitory computer-readable storage medium storing a program for causing a computer to function as an information processing apparatus comprising: a separation unit configured to separate, from an image captured by an image capturing unit included in a plurality of image capturing units configured to capture an image of an object from a plurality of viewpoints, a region of the object using a method for light applied to an image-capturing target region of the image capturing unit at a time of capturing the image; and a determination unit configured to determine a shape of the object based on the region of the object separated by the separation unit from each of a plurality of images captured by the plurality of image capturing units. 